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Abstract 


A number of school districts and states have implemented transition intervention programs 
designed to help high school students graduate ready for college. This study estimates the 
effectiveness of a transition program implemented statewide in Kentucky for high school 
seniors called Targeted Interventions (TI). Using 11 years of linked panel data, this study tracks 
the college progression of seven cohorts of students as they move from high school into 
college. Using a difference-in-regression discontinuity design, we estimate the program’s 
impact on college credit attainment and transfer as well as the extent to which the program has 
helped reshape pathways through college. We find that the TI program significantly increased 
the likelihood that students would take at least 15 credits during the first term in college, a key 
measure that has been shown to be predictive of college completion. These early effects, 
however, do not translate into statistically significant impacts on the likelihood of transfers 
from a 2-year to a 4-year college, or the likelihood of earning enough credits to graduate from 
college. We discuss some possible explanations for why the TI program did not lead to 
observable improvements in college transfer or credit accumulation. 


ill 


1. Introduction 


Many students who enroll in college do not earn a degree. Among first-time, full-time 
college students, only 30% graduate from 2-year colleges within 3 years of initial enrollment and 
only 60% graduate from 4-year colleges within 6 years.' Weak pre-college preparation is thought 
to be one of the main reasons for poor attainment (Bettinger et al, 2013). Traditionally, 
underprepared entering college students were placed in developmental education (DE) in college 
before they were able to enroll in college-level courses. However, DE has been found to be both 
costly and ineffective (e.g., Boatman & Long, 2018; Valentine, Konstantopoulos, & Goldrick- 
Rab, 2017), and some scholars and administrators consider getting students ready for college a 
shared responsibility between high school and college (Barnett, Fay, Pheatt, & Trimble, 2016).? 

In response, an increasing number of states have implemented transition intervention 
programs for high school students designed to help them graduate high school ready for college. 
Between 2013 and 2017, the number of statewide programs more than doubled, up to 17, with 
another 22 states offering local transition intervention programs (Barnett, Chavarin, & Griffin, 
2018). Transition intervention programs vary in scope, placement mechanism, delivery format, 
and whether successful completion automatically advances students to college-level coursework, 
but most include a screening assessment for college readiness, a transition curriculum for those 
deemed to need the intervention, and an exit evaluation (Fay, Barnett, & Chavarin, 2017). 

But while high school-to-college transition intervention programs are viewed (e.g., 
Barnett, Fay, & Pheatt, 2016) as holding the promise of improving student college preparedness, 
there is limited evidence (we describe this below) on the effectiveness of this type of 
' See https://nces.ed.gov/programs/raceindicators/indicator RED.asp. 

? Additional comments can be found in https://hechingerreport.org/solution-obvious-rare-making-high-school- 


graduates-ready-college/ and https://www.edweek.org/teaching-learning/transitional-courses-catch-on-as-college- 
prep-strategy/2014/02. 


intervention, with existing findings mixed at best. In this paper, we explore a statewide transition 
intervention program in Kentucky—Targeted Interventions (TI)—and its impact on longer term 
college outcomes. Kentucky is an especially interesting state to investigate because there is 
evidence (Xu et al., forthcoming) that the Kentucky TI program has reduced the need for DE of 
initially low-achieving students who attend college in the state. To our knowledge, this is the 
first study to explore longer term college outcomes, including the likelihood of transferring from 
a 2-year to a 4-year college and obtaining enough credits to earn a college degree. 

We find that TI significantly increased the likelihood that students would take at least 15 
credits during the first term in college—a key measure of early college progress that has 
been shown to be predictive of college completion (e.g., Attewell & Monaghan, 2016). But we 
find little evidence that these initial effects led to detectible improvements in measured progress 
through college, and TI did not have a statistically significant effect on the likelihood of transfers 
from a 2-year to a 4-year college or the likelihood of completing the minimum number of credits 
required for graduation (60 credits for 2-year college and 120 credits for 4-year college) within 
150% of the normal time for completion (i.e., 3 years for those who start in a 2-year college and 
6 years for those who start in a 4-year college). 

In what follows, we first summarize existing evidence about transition intervention in 
general and describe the design and implementation of Kentucky’s TI program. This is followed 
by a brief description of the data, samples, and research method. The impact of TI on various 
student outcomes is reported in Section 4, which is followed by a discussion of why TI did not 


lead to observable improvements in longer-term credit accumulation and transfer. 


2. Transition Intervention and Its Implementation in Kentucky 


2.1. Theory and Existing Evidence 

Two thirds of entering 2-year college students take at least one developmental education 
course, suggesting that completing a high school diploma does not necessarily adequately 
prepare a student for college (Chen, 2016; Bailey & Jaggars, 2016). Transition interventions are 
thought to bridge the disconnect between high school and college by using college placement 
tests and benchmarks to align expectations across education levels and to identify high school 
students for supplemental support (Fay et al., 2017). Transition interventions are often designed 
to follow the Response to Intervention (RTI) framework (Fuchs & Fuchs, 2006), using 
assessment data to identify specific areas of deficiency, inform instruction, and monitor progress. 
In doing so, transition interventions can potentially help students by targeting specific areas of 
need. 

In addition to Kentucky, state-level transition intervention programs have been evaluated 
in West Virginia (Pheatt, Trimble, & Barnett, 2016), Florida (Mokher, Leeds, & Harris, 2018), 
and Tennessee (Kane et al., 2021). Findings in those states, all based on regression discontinuity 
(RD) designs, are less promising than in Kentucky. In all three states, transition interventions 
produced either no significant effect (Florida and Tennessee) or a negative effect (West Virginia) 
on students’ likelihood of passing introductory college courses (a key college graduation 
requirement). In Tennessee, the transition intervention produced an initial effect of a 30% 
reduction in developmental math enrollment. But the effect was not the result of improved math 
skills, as the authors demonstrated using a post-intervention assessment, but due to automatic 


exemption from DE in college upon successful completion of a transition curriculum in high 


school.’ In West Virginia and Florida, transition interventions had no detectable impact on DE 
enrollment. 
2.2. Targeted Transitional Interventions (TI) in Kentucky 

In 2009, the Kentucky General Assembly found “the continuing high rates of high school 
students who require remediation at the postsecondary education level totally unacceptable and 
an unwarranted additional expense to the state.”* As a result, the state passed legislation intended 
to better align secondary and college education, including the establishment of the diagnostic 
cycle that would become TI. The postsecondary and secondary levels were charged to “develop a 
unified strategy to reduce college remediation rates by at least fifty percent by 2014 ... and 
increase the college completion rates of students enrolled in one or more remedial classes by 
three percent (3%) annually from 2009 to 2014.” 

In terms of reducing the number of students enrolled in college remediation, the program 
appears to have partially achieved this goal by reducing the need for DE. Specifically, Xu, 
Backes, Oliveira, and Goldhaber (forthcoming) found that TI reduced the likelihood that students 
would enroll in college DE by 8-10 percentage points in math in both 2- and 4-year institutions 
among students who just missed the college-ready benchmark in 11th grade. TI also increased 
the rate at which these students pass introductory college math within the first year of college by 
four percentage points in 4-year institutions. These effects, roughly equivalent to 0.10—0.20 
standard deviations, are sizeable relative to the cost of the program, estimated to be about $600 


per student (Levin et al., 2020). These positive effects are even stronger among free/reduced- 


3 The Tennessee program automatically exempting students from DE in college after successful completion is a 
common practice across transition intervention programs (Fay et al., 2017). Kentucky’s TI is one of only five (out of 
17 statewide programs) in which successful completion of a transition intervention does not automatically place 
students out of DE in college. 

4 https://web.archive.org/web/20120113051012/http://cpe.ky.gov/NR/rdonlyres/09913AFD-9097-4374-95A 1- 
10C96060D134/0/SenateBilll_ 2009RegularSession.pdf 


price lunch—eligible students, reducing enrollment in developmental math courses by 11 
percentage points and increasing the rate of passing college math by 9 percentage points by the 
end of the first year in college. 

The TI program was first implemented for high school math in 2010—11 and high school 
English in 2011-12. Using test scores from the ACT taken by all 11th grade students in 
Kentucky public schools, TI uses predetermined cutoffs (19 for math and 18 for English) to 
identify students who may not be on track to be ready for college at the end of high school, and 
provides supplemental interventions to help those students meet college readiness expectations. 
By these standards, 59% of students were deemed not on track to be ready for college-level math 
and 43% were not on track to be ready for college English (Xu et al., forthcoming).° 

Detailed student-level intervention data were not collected until the 2013—14 cohort of 
11th grade students. These data, supplemented with interviews with program administrators and 
teachers conducted by Xu et al. (forthcoming), provide information on the format of TI delivery. 
Interventions were primarily delivered in the form of transition courses (65%) and extended 
school services (22%). These transition courses could either be a standalone course or integrated 
with other existing courses. On average, intervention services were provided four times a week, 
with each session lasting 55 minutes. Most intervention teachers used a combination of self- 


developed materials and online curricula such as ALEX, Dreambox, Edgenuity, and IXL. 


> The same ACT math cutoff is used in Tennessee, where about half of students score below the cutoff. West 
Virginia uses its own state test to place students, and while Pheatt et al. (2016) do not report the share of students 
falling below the cutoff in math, a visual inspection of their density plot (Figure 3) shows well under half of students 
meeting the benchmark. 


3. Data, Measures, and Methods 


3.1. Data 

In this study, we utilize individual-level administrative records provided by the Kentucky 
Department of Education and Kentucky Council on Postsecondary Education. We focus on 
seven cohorts of students who were high school juniors between the 2008—09 and 2014—15 
school years. These cohorts span both pre- and post-treatment periods, with pre-treatment 
cohorts represented by the earliest two cohorts for math TI and the earliest three for English TI 
(because English TI was implemented one year before math). Each cohort consists of about 
43,000 students. With 11 years of linked panel data, this study can track student progress through 
college until 3 years after initial enrollment in a 2-year college for all seven cohorts of high 
school juniors and 6 years after initial enrollment among 4-year college students for four cohorts 
of high school juniors (2008-09 through 2011-12). 

High school data include ACT scores from the mandatory statewide administration in the 
spring of the 11th grade, which were used by TI to refer students for supplemental services, as 
well as student gender, race/ethnicity, free/reduced-price lunch (FRL) eligibility, and high school 
completion status. Postsecondary data cover all enrollments in Kentucky institutions (both public 
and private).° These data include records on course enrollment, grades, credits attempted and 
earned, programs of study, transfers, and completion. 

As summarized in Xu et al. (forthcoming) and reproduced as Table 1, the study 
population is predominantly white (85%), with an FRL eligibility rate of close to 50%. The 


average ACT score is 18.8 in math and 18.5 in English. Over 90% of 11th grade students with a 


® Less than 3 percent of students with ACT scores 2-3 points around the cutoffs enrolled in out-of-state colleges, 
and there is no discontinuity in the likelihood of leaving the state at the cutoffs. 


valid ACT score graduated high school on time. The rate of college enrollment immediately after 
high school graduation varied by ACT score, ranging from about 50% among students scoring 
three points below the TI cutoffs to about 65% among students scoring three points above the 
cutoffs. 
3.2. Measures 

The key measure of success for this study is earning the minimum number of credits 
required to graduate from college. While we are able to track student transfers throughout the 
study period, reliable degree completion records are not available before 2014. As a result, we 
rely on credit accumulation as a proxy measure of college completion. Specifically, we calculate 
whether a student has accumulated at least 120 credits in a 4-year college or at least 60 credits in 
a 2-year college as an indicator of students’ likely college completion status. These thresholds 
are based on a review of graduation requirements in academic catalogues published during the 
study period, and represent the minimum thresholds below which degree completion is unlikely. 
Students typically need to satisfy additional requirements such as minimum GPA and the 
completion of core credits to graduate, and some Associate of Applied Science degree programs 
require up to 68 credits. Because about 80% of students who enroll in 2-year institutions intend 
to eventually earn a bachelor’s degree, we also measure transfer from a 2-year to a 4-year 
institution as an outcome.’ 

In order to evaluate how well credit accumulation thresholds approximate actual degree 
completion, we verify the relationship between credit accumulation and degree completion using 
available completion data from 2014 and later. As expected, falling short of the threshold almost 


perfectly predicts incompletion. However, earning more credits than the thresholds does not 


7 https://cerc.tc.columbia.edu/Community-College-FAQs.html 


always guarantee graduation. For example, 99% of 2-year college students who did not 
accumulate 60 credits 3 years after initial enrollment failed to attain any degree. On the other 
hand, 34% of students who had earned at least 60 credits did not graduate or transfer to a 4-year 
institution. 

Measures of progress at critical junctures along the pathway through college can also 
help pinpoint where students may need more help. For example, research shows that many 
students start college not knowing what to study (Bettinger et al., 2013) and switch programs 
excessively in subsequent terms (Holzer & Xu, 2021). In Kentucky 2-year colleges, for example, 
27% of first-time freshman under the age of 24 enroll without a declared major; among those 
who have declared a major at the start, about 20-30% switch among six broadly defined 
programs of study (STEM, health, business, liberal arts/social sciences, other occupational fields, 
and certificate/diploma programs) (Holzer & Xu, 2021). Some switches may be made as students 
gain more information about the field and about themselves. But there is an opportunity cost, and 
uncertainty about what field to pursue may explain why one third of 2-year college students in 
our data completed more credits than required without graduating or transferring to a 4-year 
college. Transition interventions like TI have the potential to improve the efficiency of program 


choice by exposing students to college-level content and expectations. We thus measure the 


likelihood of beginning college without a declared major and the number of times a student 


switches programs in their first year of enrollment. 


Empirical evidence also suggests that carrying an adequate course load early on is a 
strong predictor of college completion and successful transfer from a 2-year institution to a 4- 
year college (e.g., Adelman, 2006; Attewell, Heil, & Reisel, 2011). National and college 


administrative data both suggest that a 15-credit course load during the first semester is a critical 


threshold that has the strongest predictive power of college attainment (Attwell & Monaghan, 
2016; Belfield, Jenkins, & Lahr, 2016; Klempin, 2014). Students attempting at least 15 credits 
during the first semester are 9 percentage points more likely to graduate (Attewell et al., 2016), 
while paying 4-14% less per credit and 9-19% less per degree in tuition and fees (Belfield et al., 
2016). TI significantly reduced the need for DE in college, and we investigate whether this has 
led to an increase in overall course load by measuring the impact of TI on the likelihood that 
students take at least 15 credits during the first term in college. 
3.3. Research Methods 

The use of predefined ACT cut scores for TI assignment lends itself to a regression 
discontinuity (RD) design. However, the use of these cut scores in an RD framework is 
complicated by these same cut scores being used at the postsecondary level for assignment to 
developmental courses upon arrival to college. Therefore, following Xu et al. (forthcoming), we 
use a difference-in-regression discontinuity (DiRD) method to estimate the effect of TI. The 
crucial assumption of the DiRD strategy is that college placement policies or any other factors 
which affect the outcomes of students just above and below the cutoffs remained unchanged over 
time for the 11th grade cohorts under consideration. Under this assumption, DiRD uses pre-TI 
cohorts to net out the college placement policy effect in order to isolate the TI effect. 
Specifically, we estimate the RD effect of falling just below the ACT cutoff for the pre- and 
post-TI periods using 

Y¥, = A) + a,B; + K(S;) + @2B; * k(S;) + &, (1) 

where Y; is the outcome for student i and k(.) is a function of the ACT score of student i, Si, that 


is centered around the cutoff (19 for math and 18 for English) such that negative values indicate 


scores below cutoff. B; is an indicator for whether student 7’s score falls below the cutoff. The 
DiRD estimate is then the difference in a, between the post period and the pre period: 
B = wPo* 7 Pr? (2) 

As shown in Figure 1, the discontinuity of TI participation is not sharp near the cutoff. A 
large percentage of students who scored below the ACT cutoffs did not participate in TI ina 
given subject (“no shows’), and many students who scored above the cutoffs did participate 
(“crossovers”).® The estimated discontinuity in program participation around the cutoff is about 
40 percentage points in both subjects. Therefore, 8 estimates the intent-to-treat (ITT) effect of TI 
on student college outcomes. Under the assumption that the introduction of TI was the only 
policy change that relied on these specific ACT cut points during the study period, DiRD 
produces an unbiased estimate of the TI effect on student outcomes. 

Our data include integer values of ACT scores and should thus be viewed as discrete 
rather than continuous. Due to this discreteness in the running variable—each point of difference 
on the ACT is equivalent to 0.17—0.20 standard deviations—we are unable to get arbitrarily close 
to the cutoff and have to rely on parametric assumptions about the relationship between the 
outcome and the ACT score near the cutoff, incurring additional uncertainty about impact 
estimates. As a result, we report Eicker-Huber—White (EHW) standard errors for impact 
estimates following McCaffrey and Bell (2002) and Kolesar and Rothe (2018). In addition, 
DiRD mitigates the uncertainty associated with parametric assumptions to the extent that the 
parametric relationship between ACT test scores and student outcomes remains unchanged 


before and after TI took effect. 


8 Surveys and interviews suggest several explanations for noncompliance (Xu et al., forthcoming). Teachers could 
consider other performance metrics in addition to ACT scores when referring students to TI. Students also had the 
option in many schools to test out of TI. And in some cases, staff and parental resistance to remediation also played 
a role in under-participation in TI. 


10 


Using the bandwidth selection procedure implemented by Calonico et al. (2020), we 
obtain optimal bandwidths for each college outcome, subject, and institution type. In line with 
prior RD studies that use ACT scores as a running variable (e.g., Boatman, 2012; Ran & Lin, 
2019), optimal bandwidths range mostly between two and five points. We also investigate the 
robustness of key findings to different bandwidths and find that the choice of bandwidth is 
generally not substantively important. Finally, given the small bandwidths, k(S;) is entered into 
equation (1) as a linear function of test scores. 

3.4. Validity 

In addition to the usual assumptions of an RD—the existence of significant discontinuity 
in treatment receipt at the cutoff, the integrity of the running variable, no differential sample 
attrition at the cutoff—DiRD also assumes that the discontinuity in student outcomes at the 
cutoff due to college placement policies would have remained constant throughout the study 
period in the absence of TI. As demonstrated in Xu et al. (forthcoming), which uses the same 
identification approach we rely on here, there is no evidence that any of these assumptions are 
violated. The discontinuity in TI participation around ACT cutoffs is about 40 percentage points 
for both subjects. Manipulation of ACT scores is unlikely since teachers and schools were not 
involved in scoring and cutoffs were predetermined. We also use scores from the spring of the 
11th grade, when students took the tests for the first time, to avoid potential self-selection issues 
stemming from test retakes. As expected, the distribution of ACT scores shows no signs of 
lumpiness through the cutoff. 

Consistent with the integrity of the running variable, key baseline covariates are 
continuous at the cutoff. Using student gender, race/ethnicity, and FRL eligibility as the 


outcome, Xu et al. (forthcoming) estimated the DiRD model as described above and found no 
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detectable discontinuity at the cutoff except for the percentage of female students, where the 
difference is equivalent to about 0.10 standard deviation; this table is reproduced as Table 2 here. 
Controlling for gender (as well as other covariates) in the DiRD model produces no meaningful 
change in estimated TI effects on college outcomes. 

As a further test for the DiRD assumption that the college placement policy effect would 
have remained constant in the absence of TI, Xu et al. (forthcoming) conducted a falsification 
test by estimating TI “effects” away from the actual cutoff. Like the math cutoff of 19, a cutoff 
of 22 has also been used by colleges in Kentucky to place students in college math courses 
(albeit at a higher level). Unlike the cutoff of 19, the cutoff of 22 was unrelated to TI assignment. 
In the absence of TI, discontinuities in student outcomes around the cutoff of 22—due to college 
placement policies—remain constant before and after the implementation of TI, and the 


estimated DiRD “effects” are null.’ 


4. Findings 

The estimated TI effects are presented in Table 3. The rows of the table represent 
different college outcomes, with Panel A consisting of early college outcomes and Panel B end- 
of-college outcomes. The estimated TI impact on each college outcome is reported by subject 
and college type as columns. For each unique regression, we show the estimated TI impact, its 
associated EHW standard error, optimal bandwidth, and sample size. In Table 4 we show the 
sensitivity of the estimates provided in Table 3 to different bandwidths and present separate pre- 
and post-intervention RD estimates of the effect of failing ACT cutoffs. 
° The results in this paper are primarily obtained from the sample of students who enrolled in college. Xu et al. 
(forthcoming) use the DiRD model to examine the effect of TI on college enrollment and finds no significant effect 


on enrolling in either a 2- or 4-year college. This suggests that any TI effects on college outcomes are not achieved 
by altering the composition of college-going students. 
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4.1. Early college outcomes 

As noted in Section 3.2 above, there are reasons to believe that TI might affect various 
early college pursuits and progress towards a degree. Estimates reported in Table 3 suggest that 
TI has marginal effects on some types of program choices. These effects are not consistent across 
subject or institution type, and some are sensitive to bandwidth choices. The findings that are 
consistent across bandwidth choices (Table 4) are that TI in math reduces the likelihood that 2- 
year college students start in a health field by 3 percentage points, and that TI in English 
increases the likelihood that 4-year college students start in a STEM field by 3 percentage points. 
TI in either subject has no effect on the frequency of program switches within the first year or 
first 3 years in college. 

During the first term, TI in math increases the likelihood of students enrolling in at least 
15 credits by 6 percentage points in 2-year colleges and 5 percentage points in 4-year colleges 
(Table 3). This is consistent with the reduced need for developmental education reported in Xu et 
al. (forthcoming), suggesting that students used the freed-up time early in their college journeys 
to take more college-level courses. Figure 2, which plots select college outcomes against ACT 
scores for the pre-treatment (circles) and post-treatment (triangles) periods, shows that the 
positive impact is achieved by eliminating the gap between students scoring just above and 
below the ACT cutoff: Before the implementation of TI, due to the same TI cutoff also being 
used to place students into developmental education in college, students scoring just below the 
ACT cutoff were significantly less likely than students scoring just above the cutoff to enroll in 
at least 15 credits during the first term (estimates are presented in Table 3). In the post- 
implementation period, by contrast, the gap in first-term course load was mostly eliminated. 


Results from Table 2 also show that this finding is robust to alternate bandwidths. TI in English 
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has weaker effects on first-term course load, increasing the likelihood of students attempting at 
least 15 credits by 4 percentage points in 4-year colleges but having no significant impact in 2- 
year colleges. !° 

4.2. Long Term Credit Accumulation and Transfer 

To assess whether the above impacts of TI on early college choices and progression led 
to changes in longer term college outcomes, we examine total credits earned and the likelihood 
of earning at least 60 credits at the end of 3 years in college or at least 120 credits after 6 years in 
a 4-year college (Table 3, Panel B). TI has no detectable effect on any of these measures. Among 
4-year college students, those who were subject to math TI appear to have completed close to 
one additional course (2.3 credits) more than students who were not subject to math TI by the 
end of 3 or 6 years in college, but these are imprecisely estimated. Results in Table 3 suggest that 
TI has no measurable impact on helping students meet the minimum credit requirement for 
graduation. TI also has no detectable effect on the likelihood that 2-year college students transfer 
to a 4-year institution. 

Figure 3 depicts the trajectory of credit accumulation by term. Cumulative credits are 
measured at the end of each fall, spring, and summer term for the first eight terms. Trajectories 
are plotted for students scoring within three points above and below the cutoff for cohorts before 
and after the implementation of TI separately. Gaps in credit accumulation between higher and 
lower scoring students emerge at the end of the first term. The higher scoring group also appears 
to earn credits at a faster rate, resulting in a steadily widening gap in completed credits over the 


course of 3 years. Notably, the gap between the higher and lower scoring groups at each point in 


'0 Tn results available from the authors, we find that TI in math is associated with attempting more STEM credits in 
the first year, which is likely a consequence of being more likely to take a credit-bearing course in math instead of 
developmental math. However, the impact on total number of STEM credits accumulated by the end of the first year 
is small and not statistically significant. 
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time appears to remain unchanged before and after TI implementation. In other words, the lack 
of TI effect on accumulated credits at the end of college is not because early gains dissipated 
later. Rather, positive TI effects on early progress did not result in discernible gains in college 
progression early on. 
4.3. Subgroup analysis 

We investigate whether TI has similar effects on college outcomes across three student 
subgroups that are thought to face unique challenges in college. FRL-eligible students, for 
instance, may be more sensitive to the labeling effect as not being ready for college (Mokher et 
al., 2018). Students who missed ACT cutoffs in more than one subject and therefore were subject 
to intervention in multiple subjects! could be overwhelmed by the burden of interventions, 
which in turn could undermine the effectiveness of TI. Finally, high schools with a high 
percentage of students who need transitional intervention'* may be overburdened by the 
prevalence of a need for intervention; students enrolled in these high schools may receive 
interventions that are less effective. 

Estimated DiRD effects of TI on college outcomes, grouped by student population, are 
reported in Table 4. They largely mirror findings among the overall student population. Math TI 
improves the likelihood of students taking a 15-credit course load in the first term by 6-13 
percentage points in 2-year colleges for all student subgroups. English TI has a similar effect on 
first-term course load among FRL-eligible students and students from high-need schools who 
later enrolled in 4-year colleges. However, math TI’s effect on early momentum is no longer 


significant among these student groups in 4-year colleges, which is partially due to smaller 


'! Half of students who failed to meet the math benchmark also participated in English remediation, and 76% of 
students who failed to meet the English benchmark received remediation in math. 
"2 Schools in the top quarter have at least 69% of their students missing the ACT cutoff in math and 54% in English. 
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sample sizes and the loss of statistical power. TI also appears to reduce the likelihood that 2-year 
college students start with short-term diploma or certificate programs by 9 percentage points for 
FRL-eligible students and 14 percentage points for students from high-need schools. We find no 


detectable TI effect on any other college outcomes. 


5. Further Exploration 


The direct goal of TI and other types of transition interventions is to reduce the need for 
DE in college. Xu et al. (forthcoming) find evidence that TI has reduced developmental course 
taking, and we show here that TI also affects other early college outcomes, including first-year 
course load. However, we do not find evidence that these initial gains put students on an 
accelerated trajectory towards long-term credit accumulation or successful transfer from 2- to 4- 
year colleges. In this section, we explore two reasons that might explain the lack of statistical 
significance for long-term college outcomes despite evidence that TI decreases the likelihood 
that students are deemed to need development (non-credit-bearing courses) in college. 

The first potential explanation is that intervention activities were no more beneficial for 
long-run college success than the regular high school courses they crowded out. Under this 
hypothesis, the only channel through which TI would affect long-run college outcomes would be 
by moving some students out of developmental courses and into introductory courses early in 
their college paths, but not by increasing overall skills associated with college progression. 
Tabulations using transcript data show that the most frequently taken math courses by students 
not subject to TI were Algebra II, Algebra III, and College Algebra. Taking Algebra II or higher 
is considered a critical precollegiate milestone for eventually earning a college degree (Adelman, 


2006). By contrast, student-level intervention data show that math TI most frequently focused on 
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algebraic thinking and math reasoning. Although the definition of these content areas is 
unknown, similar terms were also used in course descriptions for developmental math courses in 
college.'* This course substitution—in contrast to, for example, increasing total learning time— 
may impose an upper limit on the plausible long-run college impacts of TI. We explore this 
mechanism in three different ways: by measuring crowd-out of high school courses directly 
(Section 5.1), by examining college outcomes for a subset of successful TI completers (Section 
5.2), and by examining a similar intervention earlier in secondary school (Section 5.3). 

The second explanation is that given the effect size on early college outcomes and the 
strength of the relationship between early college outcomes and later college outcomes estimated 
in previous studies, the expected long-run effects of TI are quite small. Thus, we do not have the 
power to detect the expected long-run effects of TI. We provide this discussion in Section 5.4. 
5.1. Crowd-Out of Other High School Courses 

We begin by investigating the extent to which TI may have crowded out regular high 
school courses. Although some TI students received interventions as extended school services, 
most interventions were delivered as transition courses during the normal school day. Time 
students spend on transition interventions during the normal school day implies less time spent 
on what students otherwise would have been doing. 

To investigate how course-taking changed after the introduction of TI for students below 
the cutoff relative to above the cutoff, we divide courses into four categories based on coding 
conventions stipulated by the Kentucky Uniform Academic Course Codes: transition courses in 


math or English, regular non-transition courses in math or English, career technical education 


'3 See, for example, the course description for Mathematical Literacy (MAT 075) in the academic catalogue of the 
Kentucky Community and Technical College System. 
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(CTE) courses, and other non-transition courses.'* Results are presented in Table 6 in the same 
format as in Tables 3 and 5, with each coefficient representing the DiRD estimate on a given 
outcome for a given subject. As expected, we see that the introduction of TI led students below 
college readiness benchmarks to take more transition math courses by 12 percentage points and 
more transition English courses by 2 percentage points. For math TI, we also see evidence that 
the increase in transition math course-taking largely came at the expense of regular math courses 
(an 8-percentage-point drop in enrollment) rather than CTE courses or other courses, although 
the latter is very imprecisely estimated. For English, we see smaller reallocation effects. 
5.2. Substitution of Curriculum 

As another test of whether TI content was geared more toward the skills needed to avoid 
DE in college than toward the more advanced skills needed for later progression through college, 
we examine the relationship between successful TI completion and later college outcomes. In 
particular, we estimate the difference in college outcome Y between successful completers of TI 
(S_7T) and students who failed to complete TI (F'_77) relative to students who did not receive any 
intervention, the reference group, in the following OLS regression: 

Y¥; = Bo + Bi S_TI, + Bo F_TI, + COV +O +S, + uj. (3) 

Equation (3) also controls for ACT score fixed effects S and a vector of covariates COV that 
include student race/ethnicity, gender, FRL eligibility, and whether a student was eligible for TI 
in more than one subject. 

Estimates reported in Table 7 show that among students with the same ACT score and 


similar background characteristics, successful completion of TI is associated with lower 


'4 We examine CTE course-taking as a separate course category because low-achieving students are more likely to 
focus on CTE, so there is significant overlap between the target student population of transition interventions and 
students who are likely to benefit from CTE. 
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likelihood of DE enrollment, but not with other measures of progress later on (i.e., the likelihood 
of passing credit-bearing college courses during the first year, taking at least 15 credits in the 
first term, or completing at least 60 credits by the end of three years). Unsuccessful exit from TI, 
on the other hand, is associated with higher likelihood of DE enrollment and weaker college 
outcomes. These findings are not causal due to selection into TI and into completion of TI, but 
taken together, they suggest that the standards used by high schools to judge student progress 
towards college readiness are largely consistent with the skills needed to place out of 
developmental courses, but not sufficient to better prepare students for college-level instruction. 
5.3. Intervention and Student Skills 

As an additional test for whether TI might be plausibly expected to lead to measurable 
changes in college-ready skills, we examine the relationship between falling below the Grade 8 
cutoff (and thus being discontinuously likely to receive TI) and ACT scores in 11th grade. While 
this paper focuses on Grade 12 interventions because this is the grade that the TI program has 
been continuously focusing on, TI was implemented as early as Grade 8 for a short period 
between 2012-13 and 2014-15. Using these Grade 8 cohorts allows for a direct test of student 
knowledge gains due to TI that we cannot perform for Grade 12 interventions due to a lack of 
access to post-intervention test scores. If TI did not lead to an appreciable change in long-run 
student skills, we might expect to find little relationship between TI in Grades 8—9 and outcomes 
years later. 

Because reporting was not mandatory for interventions in the eighth and ninth grades, we 


are limited to examining the impact of TI among “complier” districts where reported TI 
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participation is strongly associated with failing the cutoffs.'° As shown in Figure 4, students in 
these districts participated in TI discontinuously at the cutoffs. The estimated first stage (Table 8) 
finds that students scoring just below the cutoffs were 40—60 percentage points more likely than 
students scoring just above the cutoffs to receive interventions in a given subject. 

We use five cohorts of eighth grade students to estimate the ITT effect using the same 
DiRD model as in our main analysis, with 2014-15 representing the post-intervention period and 
2009-2012 the pre-intervention period. Results are shown in Table 9. While estimates for some 
outcomes are imprecise due to the limitations noted above, we do not find any evidence that 
falling just below the college-readiness threshold in eighth grade led to students “catching up” by 
meeting ACT benchmarks in 11th grade, or, further down the road, taking credit-bearing math or 
English courses. In particular, we are able to rule out a positive effect on ACT math scores any 
larger than 0.80 ACT points (-0.19 + 1.96 * 0.51), and the point estimates on ACT scores and the 
likelihood of meeting ACT benchmarks are negative. While this is consistent with TI not moving 
the needle on long-run student outcomes, possibly due to the lack of an increase in overall 
instruction time, there are two important caveats: First, it is possible that the ““complier’” districts 
are unrepresentative of districts in the state. Second, this program was terminated only 3 years 


later, and it is possible it was not implemented with high fidelity. 


5.4. Statistical power 
Although Kentucky’s transition intervention has strengthened early college outcomes that 


are shown to be predictive of later attainment, the effects may be too small to produce 


'S We use equation (1) to estimate the discontinuity of reported TI participation around EXPLORE cutoffs district by 
district. The optimal bandwidth is 3 for math and 4 for English. Districts with an estimated discontinuity of 0.20 and 
at least 50 students within bandwidth were selected. This results in nine districts for math and six districts for 
English. We also used alternative criteria for district selection (i.e., a discontinuity of 0.15 and 0.25, with and 
without sample size restrictions, and alternating the bandwidth), but the findings on earlier intervention’s impact on 
student outcomes did not materially change. 
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statistically detectable changes in credit accumulation and transfer rates. Take first-term course 
load as an example. The empirical literature on early progress in college has found that taking 15 
or more credits during the first term increases the likelihood of college completion by 9 
percentage points (Attewell et al., 2016). Since TI in math is estimated to have increased the 
likelihood that students take at least 15 credits by 5—6 percentage points, a direct extrapolation 
suggests that we can expect the average graduation rate to increase by roughly half of a 
percentage point (1.e., 0.09 x 0.05 ~ 0.005). Converting the percentage point change to an effect 


size using Cox index conversion (Sanchez-Meca, Marin-Martinez, & Chacon-Moscoso, 2003): 


LOR 
Or > 
1.65 


Acox = 


3 
4N-9 


where w = [1 — ] with N denoting total sample size, LOR = In(Odds;) — In (Odds;), and 


assuming 30% of students graduate before the intervention, we estimate the expected impact of 
TI on college graduation rate to be roughly equivalent to 0.01 standard deviations. Given a 
sample size of 54,000 students within a bandwidth of 3 around the cutoff who are about evenly 
split between the pre- and post-TI samples, a typical RD design is expected to have a minimum 
detectable effect size (MDES) of 0.05 standard deviations. '® The estimated MDES for a DiRD 
design, assuming independence between the pre- and post-TI samples, would be V2 * 0.05 = 
0.07 standard deviations. Thus, the research design does not have sufficient power to detect a 
change of 0.01 standard deviations (i.e., a 0.5 percentage point change in graduation rate). 
Even if a study were powered to detect a 0.5 percentage point change—which would 


require at least 2 million observations split between the pre- and post-intervention periods—it 


‘6 MDES is estimated following Schochet (2008) for a two-tailed test with a significance level of 0.05 and statistical 
power of 0.80. Estimation assumes the following parameters: Students within a bandwidth of 3 around the cutoff are 
evenly divided between treatment and control groups. Covariates explain 15 percent of the variation in the outcome 
variable. 
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would still raise the question of whether such a relatively small effect is substantively important. 
The Kentucky Council for Postsecondary Education (CPE) has set out a goal of raising the 
percentage of Kentuckians with a postsecondary degree from 45.5% in 2017 to 60% in 2030 
(CPE, 2019). Thus, even if it led to a 0.5 percentage point increase in the rate at which students 
earned a postsecondary degree, TI alone would not be sufficient to move the needle towards this 


college completion goal. 


6. Conclusion 


In this study, we investigate the potential impact of Kentucky’s high school transition 
intervention program on the likelihood of college completion and transfer as well as on some of 
the key predictors of college success. The key finding is that the TI program achieved its direct 
goal by reducing the number of students enrolling in developmental education in college, which 
in turn allowed more students to build early momentum by taking 15 or more credits during the 
first term. However, the early benefits of the TI program did not lead to detectable improvements 
in measured proxies of college attainment. 

As discussed in Section 5, one possible explanation for the findings is that time 
constraints led to the substitution of regular high school courses with intervention activities and 
as a result, there was no increase in the time that students spent on academic pursuits in high 
school. Because the immediate mandate for TI was to get students ready to access “credit- 
bearing coursework without the need for developmental education or supplemental courses” in 
college (CPE, 2010, p. 7), it is not surprising that TI chose to focus on skills required to place out 
of DE in college, explaining the changes in early college outcomes. However, the transition 


curriculum, while helping students avoid the need for college developmental courses, does not 
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appear to have helped a measurable share of students to develop the necessary skills to progress 
through college relative to what they would have otherwise taken. 

Kentucky’s strategic agenda for postsecondary education set out a goal of increasing the 
share of Kentuckians with a postsecondary degree from 44% in 2015 to 60% in 2030.!’ If much 
of this increase is going to come from students who currently do not appear prepared for college, 
our evidence suggests that more intensive intervention may be required. Although no direct 
evidence is available about the relationship between the “dosage” of transition intervention and 
college outcomes, empirical findings from elementary and middle school grades consistently 
suggest that increased instruction time has positive effects on student test scores (e.g., Taylor, 
2014; Cortes, Goodman, & Nomi, 2015; Figlio, Holden, & Ozek, 2018). While we cannot 
definitively say whether TI’s lack of statistically identifiable impacts on longer-run college 
outcomes is because of substitution among high school courses or because the effect is too small 
to detect, it appears that high school-to-college transition interventions that do not increase total 
instruction time have a limited scope for impact on long-run college success. 

In addition to expanded opportunities for students to acquire the requisite academic skills 
for college, another potential way to increase the impact of transition intervention programs is to 
add support for non-academic college readiness. Students on the margin of being deemed 
academically ready for college face so many additional challenges that they are unlikely to 
complete college in the first place: For example, only about 1 in 4 students near the ACT college 
readiness math cut score obtain 60 credits within 3 years in 2-year institutions (Figure 2). These 
additional challenges include behavioral, informational, and financial constraints (Page & Scott- 


Clayton, 2016). These barriers often overlap, and academically vulnerable students could benefit 


'7 http://cpe.ky.gov/ourwork/60x30.html 
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from non-academic support (Karp, 2011). Our analysis shows, for example, that many students 
found it difficult to decide on what to study in college, and that TI did not reduce either the 
prevalence of students starting college with an undeclared program of study or excessive 
program switching in subsequent years. This suggests potential efficiency to be gained if 
students were able to make informed choices early on, and high school transition interventions 
could help by providing information on the values of majors in the labor market, personalized 


advising, and career counseling. 


24 


References 


Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through 
college. Washington, DC: U.S. Department of Education. 

Attewell, P., Heil, S., & Reisel, L. (2011). Competing explanations of undergraduate 
noncompletion. American Educational Research Journal, 48(3), 536-559. 

Attewell, P., & Monaghan, D. (2016). How many credits should an undergraduate take?. 
Research in Higher Education, 57(6), 682-713. 

Bailey, T. & Jaggars, S. (2016). When college students start behind. The Century Foundation. 

Barnett, E. A., Chavarin, O., & Griffin, S. (2018). Math Transition Courses in Context: 
Preparing Students for College Success. CCRC Research Brief. New York, NY: 
Community College Research Center, Teachers College, Columbia University. 

Barnett, E. A., Fay, M. P., & Pheatt, L. (2016). Implementation of high school-to-college 
transition courses in four states. New York, NY: Columbia University, Teachers College, 
Community College Research Center. 

Barnett, E. A., Fay, M. P., Pheatt, L., & Trimble, M. J. (2016). What we know about transition 
courses. New York, NY: Columbia University, Teachers College, Community College 
Research Center. 

Baumert, J., Nagy, G., & Lehmann, R. (2012). Cumulative advantages and the emergence of 
social and ethnic inequality: Matthew effects in reading and mathematics development 
within elementary schools?. Child Development, 83(4), 1347-1367. 

Belfield, C., Jenkins, D., & Lahr, H. (2016). Momentum: The academic and economic value of a 


15-credit first-semester course load for college students in Tennessee. CCRC Working 


25 


Paper No. 88. New York, NY: Community College Research Center, Teachers College, 
Columbia University. 

Bettinger, E. P., Boatman, A., & Long, B. T. (2013). Student supports: Developmental education 
and other academic programs. The Future of Children, 23(1): 93-115. 

Boatman, A. (2012). Evaluating institutional efforts to streamline postsecondary remediation: 
The causal effects of the Tennessee Developmental Course Redesign Initiative on early 
student academic success. New York, NY: National Center for Postsecondary Research, 
Columbia University. 

Boatman, A., & Long, B. T. (2018). Does remediation work for all students? How the effects of 
postsecondary remedial and developmental courses vary by level of academic 
preparation. Educational Evaluation and Policy Analysis, 40(1), 29-58. 

Calonico, S., Cattaneo, M. D., & Farrell, M. H. (2020). Optimal bandwidth choice for robust 
bias-corrected inference in regression discontinuity designs. The Econometrics Journal, 
23(2), 192-210 

Chen, X. (2016). Remedial Coursetaking at U.S. Public 2- and 4-Year Institutions: Scope, 
Experiences, and Outcomes (NCES 2016-405). U.S. Department of Education. 
Washington, DC: National Center for Education Statistics. 

Cortes, K. E., Goodman, J. S., & Nomi, T. (2015). Intensive math instruction and educational 
attainment long-run impacts of double-dose algebra. Journal of Human Resources, 50(1), 
108-158. 

Council on Postsecondary Education. (2010). Unified strategy for college and career readiness: 


Senate Bill 1 (2009). Retrieved from 


http://education.ky.gov/educational/CCR/Documents/CCRUnifiedPlan_draft.pdf. 


26 


Council on Postsecondary Education (CPE). (2019). Progress report: Kentucky’s progress 
toward postsecondary strategic agenda goals. Frankfort, KY: Author. Retrieved from 
http://cpe.ky.gov/data/reports/2019progressreport.pdf. 

Fay, M. P., Barnett, E., & Chavarin, O. (2017). How states are implementing transition 
curricula: Results from a national scan. New York, NY: Columbia University, Teachers 
College, Community College Research Center. 

Figlio, D., Holden, K. L., & Ozek, U. (2018). Do students benefit from longer school days? 
Regression discontinuity evidence from Florida's additional hour of literacy instruction. 
Economics of Education Review, 67, 171-183. 

Fuchs, D., & Fuchs, L. S. (2006). Introduction to response to intervention: What, why, and how 
valid is it?. Reading research quarterly, 41(1), 93-99. 

Geary, D.C., Hoard, M. K., Nugent, L., & Bailey, H. D. (2013). Adolescents’ functional 
numeracy is predicted by their school entry number system knowledge. PLoS ONE, 8(1): 
054651. 

Holzer, H. & Xu, Z. (2021). Community College Pathways for Disadvantaged Students. 
Community College Review. Advanced online publication. 
https://doi.org/10.1177/00915521211002908. 

Kane, T. J., Boatman, A., Kozakowski, W., Bennett, C., Hitch, R., & Weisenfeld, D. (2021). Is 
college remediation a barrier or a boost? Evidence from the Tennessee SAILS 
program. Journal of Policy Analysis and Management. Advanced online publication. 
https://doi.org/10.1002/pam.22306. 

Karp, M. M. (2011). Toward a New Understanding of Non-Academic Student Support: Four 


Mechanisms Encouraging Positive Student Outcomes in the Community College. CCRC 


27 


Working Paper No. 28. New York, NY: Community College Research Center, Teachers 
College, Columbia University. 

Klempin, S. (2014). Redefining full-time in college: Evidence on 15-credit strategies. New York, 
NY: Community College Research Center, Teachers College, Columbia University. 

Kolesar, M., & Rothe, C. (2018). Inference in regression discontinuity designs with a discrete 
running variable. American Economic Review, 108(8), 2277-2304. 

Levin, J., Carbuccia, M., Adelman-Sil, E., & Danks. A. (2020). Kentucky targeted intervention 
program cost study. Washington, DC: American Institutes for Research. 

Magnuson, K. A., & Duncan, G. J. (2006). The role of family socioeconomic resources in the 
black—white test score gap among young children. Developmental Review, 26(4), 365- 
399: 

McCaffrey, D. F., & Bell, R. M. (2002). Bias reduction in standard errors for linear and 
generalized linear models with multi-stage samples. In Proceedings of Statistics Canada 
Symposium (pp. 1-10). 

Mokher, C., Leeds, D., & Harris, J. (2018). Adding it up: How the Florida college and career 
readiness initiative impacted developmental education. Educational Evaluation and 
Policy Analysis, 40(2), 219-242. 

Page, L. C., & Scott-Clayton, J. (2016). Improving college access in the United States: Barriers 
and policy responses. Economics of Education Review, 51, 4-22. 

Pheatt, L. E., Trimble, M. J., & Barnett, E. (2016). Improving the transition to college: 
Estimating the impact of high school transition courses on short-term college outcomes. 
New York, NY: Community College Research Center, Teachers College, Columbia 


University. 


28 


Ran, F. X., & Lin, Y. (2019). The effects of corequisite remediation: Evidence from a statewide 
reform in Tennessee [Community College Research Center Working Paper 115]. New 
York, NY: Teachers College, Columbia University. 

Roderick, M., Nagaoka, J., and Allensworth, E. (2006). From high school to the future: a first 
look at Chicago public school graduates’ college enrollment, college preparation, and 
graduation from four year college. Chicago: The University of Chicago Consortium on 
Chicago School Research. 

Stock, James H., and Motohiro Yogo. 2005. “Testing for Weak Instruments in Linear IV 
Regression.” Pages 80-108 in Donald W. K. Andrews and James H. Stock (eds.), 
Identification and Inference for Econometric Models: Essays in Honor of Thomas 
Rothenberg. New York: Cambridge University Press. 

Scott-Clayton, J., & Rodriguez, O. (2015). Development, discouragement, or diversion? New 
evidence on the effects of college remediation policy. Education Finance and Policy, 
10(1), 4-45. 

Sanchez-Meca, J., Marin-Martinez, F., & Chacon-Moscoso, S. (2003). Effect-size indices for 
dichotomous outcomes in meta-analysis. Psychological Methods, 8(4), 448-467. 

Schochet P. Z. (2008). Statistical power for random assignment evaluations of education 
programs. Journal of Educational and Behavioral Statistics, 33(1), 62-87. 

Taylor, E. (2014). Spending more of the school day in math class: Evidence from a regression 
discontinuity in middle school. Journal of Public Economics, 117, 162-181. 

Valentine, J. C., Konstantopoulos, S., & Goldrick-Rab, S. (2017). What happens to students 
placed into developmental education? A meta-analysis of regression discontinuity 


studies. Review of Educational Research, 87(4), 806-833. 


29 


Xu, Z., Backes, B., Oliveira, A., & Goldhaber, D. (forthcoming). Targeted Interventions in High 


School: Preparing Students for College. Educational Evaluation and Policy Analysis. 


30 


Figures and Tables 


Figure |. Targeted Intervention (TI) Participation Rate by ACT Score, by Subject. 
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Note: Based on ACT math and English tests taken by high school juniors in 2014, 2015, and 2016. Test scores are 


centered around 19 for math and 18 for English. 
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Figure 2. College Outcomes by ACT Test Scores, by Subject and Institution Type 
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Figure 3. Cumulative Credits Earned by Term 


Credits earned by term in college 
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Figure 4. Intervention Participation Rate by Eighth Grade EXPLORE Score, Select School 
Districts, 2014-15 


First-stage discontinuity, grade 8-9 intervention 
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Table 1. Descriptive Statistics of Study Samples: 2009-2016 (reproduced from Xu et al., 


forthcoming) 
Math RD Sample English RD Sample 
Full Sample (ACT-M 16-21) (ACT-E 15-20) 
Demographics (%) 
Female 50 52 51 
White 84 85 85 
Black 10 10 10 
Hispanic 5 5 5 
FRL 48 50 51 
Test Scores (Mean and 
standard deviations) 
ACT Math 18.8 17.6 17.7 
(4.5) (1.6) (2.8) 
ACT Reading 19.4 18.7 18.1 
(5.9) (4.6) (3.6) 
ACT English 18.5 17.9 17.4 


(6.3) (4.6) (1.8) 


Table 2. Covariate Balance Check (reproduced from Xu et al., forthcoming) 


Math English 

All College 2 year 4 year All College 2 year 4 year 

Female -0.04*** = -0.05*** = -0.07** — -0.04* 0.01 0.01 0.04 -0.02 
(0.02) (0.02) (0.03) (0.03) (0.02) (0.02) (0.03) (0.03) 

White 0.01 0.01 -0.02 0.03* 0.00 -0.00 0.01 -0.01 
(0.01) (0.01) (0.02) (0.02) (0.01) (0.02) (0.02) (0.02) 

Black -0.01 -0.01 0.01 -0.02 -0.01 -0.02 -0.02 -0.01 
(0.01) (0.01) (0.01) (0.02) (0.01) (0.01) (0.01) (0.02) 

Hispanic 0.00 -0.00 0.00 -0.01 -0.01* -0.01 -0.02 0.00 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 

FRL -0.01 -0.01 -0.01 0.00 -0.01 -0.02 -0.04 -0.00 
(0.02) (0.02) (0.03) (0.02) (0.02) (0.02) (0.03) (0.03) 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber- 


White standard errors in parentheses. Difference-in-RD estimates assume linear association between the 


running variable and outcomes with a bandwidth of three. Results are robust to other bandwidth choices. “All” 


column contains sample of all cohorts with ACT scores, “College” a sample of students who enrolled in 


college the year after graduating high school, “2 year” a sample of students who enrolled in a 2-year institution 
the year after graduating high school, and “4 year” a sample of students who enrolled in a 4-year institution the 


year after graduating high school. 
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Table 3. Estimated Effect of Targeted Interventions (TI) on College Outcomes 


Math English 
2 year 4 year 2 year 4 year 
Coefficient/ Bandwidth/ | Coefficient/ Bandwidth/ | Coefficient/ Bandwidth/ | Coefficient/ Bandwidth/ 
Outcome S.E. N S.E. N S.E. N S.E. N 
Panel A: Early college outcomes 
Started out in a non-degree program -0.07** 3 0.01 4 
(0.03) 20304 (0.02) 20195 
Started with undeclared program 0.02 4 -0.03* 4 -0.00 5 0.00 3 
(0.04) 27473 (0.02) 32380 (0.02) 23406 (0.02) 17840 
Started in STEM 0.02* 5 0.01 5 -0.00 4 0.03** 4 
(0.01) 54400 (0.01) 66175 (0.01) 35707 (0.02) 41895 
Started in Health -0.03* 4 0.01 5 0.03* 5 0.00 4 
(0.02) 48470 (0.01) 66175 (0.01) 41648 (0.01) 41895 
Program switch frequency within | year 0.01 6 -0.00 5 -0.00 4 -0.01 4 
(0.01) 57639 (0.01) 66175 (0.01) 35707 (0.01) 41895 
Program switch frequency within 3 years 0.00 6 -0.02 5 -0.00 5 0.03 5 
(0.04) 57639 (0.04) 66175 (0.03) 41648 (0.04) 50708 
Attempted >15 credits in first term 0.06*** 5 0.05** 4 0.02 5 0.04* 4 
(0.02) 54400 (0.02) 57524 (0.01) 41648 (0.02) 41895 
Panel B: End-of-college Outcomes 
Credits earned at the end of 3 years 1.08 3 2.33 3 -0.36 5 -1.37 4 
(1.81) 36637 (1.75) 46036 (1.20) 41648 (1.62) 41895 
Earned >=60 credits by year 3 -0.00 5 0.03 3 0.01 4 -0.01 4 
(0.02) 54400 (0.02) 46036 (0.02) 35707 (0.02) 41895 
Transfer within 3 years 0.02 5 0.02 5 
(0.02) 54398 (0.01) 41646 
Credits earned at the end of 6 years 2.34 3 0.17 5 
(2.73) 46036 (2.24) 50708 
Earned >=120 credits by year 6 0.04 3 -0.02 5 
(0.02) 27336 (0.02) 29889 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates 


assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth 


selection. N is the number of students. 
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Table 4. Robustness of Results to Alternate Bandwidths 


Math English 
2 year 4 year 2 year 4 year 
BW BWt+1l BW+2 BW BW +1 BW +2 BW BW+1 BW +2 BW BWt+1 BW+2 
Attempted 315 credits in first term 
Disc. pre -0.07*** = -0.07*** — -0.07*** | -0.03*  — -0.04** — -0.05*** | -0.03*** = -0.03*** — -0.04*** -0.02 -0.03* = -0.03** 
(0.01) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.01) (0.01) (0.02) (0.02) (0.02) 
Disc. post -0.01* = -0.02*** = -0.02*** 0.01 0.01 0.00 -0.01 -0.01 -0.01 0.02 0.01 0.01 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
DifferenceinRD 0.06*** = 0.05*** = 0.04*** | 0.05** — 0.05** 0.05*** 0.02 0.02* 0.03** 0.04* 0.04** 0.04** 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.01) (0.01) (0.02) (0.02) (0.02) 
Earned >=60 credits by year 3 
Disc. pre 0.01 0.01 0.01 -0.03 -0.02 -0.02 -0.02 -0.02 -0.02 0.00 -0.01 -0.01 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) 
Disc. post 0.01* 0.00 0.00 -0.00 0.01 0.01 -0.01 -0.01 -0.01 -0.00 0.01 0.00 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Difference in RD -0.00 -0.00 -0.00 0.03 0.02 0.03 0.01 0.01 0.01 -0.01 0.01 0.01 
(0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) 
Started with undeclared program 
Disc. pre -0.03 -0.02 -0.03 0.02 0.02 0.02 -0.00 0.01 0.01 0.01 0.01 0.02 
(0.03) (0.03) (0.03) (0.02) (0.02) (0.01) (0.02) (0.02) (0.02) (0.02) (0.01) (0.01) 
Disc. post -0.01 -0.01 -0.01 -0.01 -0.00 -0.00 -0.01 -0.00 -0.01 0.01 0.01 0.01* 
(0.01) (0.01) (0.01) (0.01) (0.00) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Difference in RD 0.02 0.01 0.02 -0.03* -0.02 -0.02 -0.00 -0.01 -0.01 0.00 -0.00 -0.01 
(0.04) (0.03) (0.03) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) 
Start with STEM 
Disc. pre -0.01 -0.01 -0.01 0.00 0.00 0.00 0.00 0.00 -0.00 -0.01 -0.01 -0.01 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Disc. post 0.01** 0.01*** =0.01*** | 0.01** O0.01*** 0.01*** -0.00 0.00 0.00 0.02** 0.02***  0.02*** 
(0.00) (0.00) (0.00) (0.01) (0.00) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Difference in RD 0.02* 0.02 0.02* 0.01 0.01 0.01 -0.00 0.00 0.00 0.03** 0.03**  0.03*** 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.02) (0.01) (0.01) 
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Start with health 


Disc. pre 0.03 0.03* 0.02 -0.01 -0.01 -0.01 -0.02 -0.01 0.00 -0.00 -0.01 -0.01* 
(0.02) (0.02) (0.02) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Disc. post -0.01 -0.01 -0.00 0.00 0.00 0.00 0.01 0.01* 0.01 0.00 0.00 0.00 
(0.01) (0.01) (0.01) (0.00) (0.00) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Differencein RD — -0.03* -0.04** -0.03* 0.01 0.01 0.02 0.03* 0.02 0.00 0.00 0.01 0.02* 
(0.02) (0.02) (0.02) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Started out in a non-degree program 
Disc. pre 0.06** 0.03 0.02 -0.01 -0.01 -0.01 
(0.03) (0.02) (0.02) (0.02) (0.02) (0.01) 
Disc. post -0.00 -0.01 -0.01 -0.00 -0.00 -0.01 
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01) 
Difference in RD — -0.07** -0.04 -0.03 0.01 0.01 0.00 
(0.03) (0.02) (0.02) (0.02) (0.02) (0.02) 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates 
assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth 
selection. N is the number of students. BW+1 and BW+2 columns display results from wider choses of bandwidth. “Discontinuity pre” represents the estimated discontinuity in the 


pre period, “Discontinuity post” the different in the post period, and “Difference in RD” the difference between the two, which are the results displayed in other tables. 


39 


Table 5. Estimated Effect of Targeted Interventions (TI) on College Outcomes, by Student and School Characteristics 


Math English 
2 year 4 year 2 year 4 year 
Coefficient/ Coefficient/ Coefficient/ Coefficient/ 
Standard Bandwidth/ Standard Bandwidth/ Standard Bandwidth/ Standard Bandwidth/ 
error N error N error N error N 
Students who missed ACT cutoff in addition to subject 
Attempted >=15 credits in first term 0.07*** 5 0.03 4 0.02 5 0.03 4 
(0.02) 39206 (0.03) 29630 (0.01) 35681 (0.02) 31656 
Earmed >=60 credits by year 3 0.04 5 -0.05 3 0.01 4 -0.02 4 
(0.03) 39206 (0.03) 23725 (0.02) 31168 (0.02) 31656 
Started with undeclared program 0.05 4 -0.03 4 0.00 5 -0.01 3 
(0.04) 21217 (0.02) 18908 (0.03) 20806 (0.02) 14846 
Started out in a non-degree program -0.05 3 0.01 4 
(0.03) 15037 (0.02) 18162 
Started in STEM 0.02 5 0.01 5 -0.00 4 0.03* 4 
(0.02) 39206 (0.02) 32498 (0.01) 31168 (0.02) 31656 
Started in health -0.04** 4 0.02 5 0.02 5 0.02 4 
(0.02) 34917 (0.02) 32498 (0.02) 35681 (0.01) 31656 
Free/reduced-price lunch eligible students 
Attempted >=15 credits in first term 0.06** > 0.04 4 0.01 >) 0.06* 4 
(0.03) 26589 (0.04) 21690 (0.02) 19618 (0.04) 16070 
Earned >=60 credits by year 3 0.02 5 0.03 3 0.02 4 0.03 4 
(0.03) 26589 (0.04) 17260 (0.03) 16728 (0.03) 16070 
Started with undeclared program 0.04 4 -0.04 4 -0.03 5 -0.01 3 
(0.06) 13665 (0.03) 12811 (0.04) 11304 (0.03) 7213 
Started out in a non-degree program -0.09* 3 -0.03 4 
(0.05) 9695 (0.03) 9669 
Started in STEM 0.03 5 0.02 5 -0.00 4 0.04 4 
(0.02) 26589 (0.02) 24469 (0.02) 16728 (0.02) 16070 
Started in health -0.02 4 0.02 5 0.03 5 0.01 4 
(0.03) 23428 (0.02) 24469 (0.02) 19618 (0.02) 16070 
Top quartile school in % students below cutoff 
Attempted >=15 credits in first term 0.13*** 5 0.07 4 0.01 5 0.11** 4 
(0.04) 16362 (0.05) 12656 (0.03) 9965 (0.05) 9497 
Earned >=60 credits by year 3 0.01 5 -0.06 3 -0.03 4 -0.03 4 
(0.04) 16362 (0.05) 10041 (0.04) 8541 (0.04) 9497 


Started with undeclared program 
Started out in a non-degree program 
Started in STEM 

Started in health 


Male 
Attempted >=15 credits in first term 


Earmed >=60 credits by year 3 
Started with undeclared program 
Started out in a non-degree program 
Started in STEM 

Started in health 


Female 
Attempted >=15 credits in first term 


Earned >=60 credits by year 3 
Started with undeclared program 
Started out in a non-degree program 
Started in STEM 


Started in health 


-0.05 
(0.07) 
-0.14** 
(0.06) 
0.02 
(0.02) 
0.00 
(0.03) 


0.06** 
(0.03) 
0.00 
(0.03) 
0.05 
(0.05) 
-0.08* 
(0.04) 
0.05** 
(0.03) 
0.00 
(0.02) 


0.07**# 
(0.03) 
-0.00 
(0.03) 

0.01 
(0.05) 
-0.06* 
(0.03) 

-0.01 
(0.01) 

-0.07*# 

0.03 


30810 
5 
30810 
4 
14864 
3 
10837 
5 
30810 
4 
27587 


0.01 
(0.05) 


-0.01 
(0.03) 
0.02 
(0.02) 


0.01 
(0.03) 
0.09** 
(0.04) 
-0.01 
(0.03) 


-0.03 
(0.02) 
0.00 
(0.01) 


0.07** 
(0.03) 
-0.00 
(0.03) 

-0.05** 
(0.02) 


0.04** 
(0.01) 
0.02 
0.02 


39649 
5 
39649 


0.01 
(0.05) 
0.01 
(0.05) 
0.00 
(0.02) 
0.03 
(0.03) 


0.01 
(0.02) 
0.02 
(0.03) 
0.03 
(0.04) 
0.04 
(0.04) 
0.00 
(0.03) 
0.01 
(0.02) 


0.03* 
(0.02) 
0.01 
(0.03) 
-0.03 
(0.03) 
-0.03 
(0.02) 
-0.01 
(0.01) 
0.04* 
0.02 


23469 
4 
20158 
5 
12696 
4 
11027 
4 
20158 
5 
23469 


0.03 
(0.04) 


0.06** 
(0.03) 
-0.01 
(0.02) 


0.05 
(0.03) 
-0.00 
(0.03) 
0.00 
(0.03) 


0.04 
(0.03) 
0.02** 
(0.01) 


0.03 
(0.03) 
-0.01 
(0.03) 
-0.00 
(0.03) 


0.02 

(0.02) 
-0.02 
0.02 


22635 
4 
22635 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates 
assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth 


selection. N is the number of students. 
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Table 6. Effect of Targeted Interventions (TI) on 12th Grade Course-Taking, by Intervention 


Subject 
Math English 
Coefficient/ Bandwidth/ | Coefficient/ | Bandwidth/ 
S.E. N S.E. N 
Within-subject substitution 
Transition courses in subject 0.12*** 2 0.02*** 3 
(0.02) 93,243 (0.01) 114,410 
Non-transition courses in subject -0.08*** 2 -0.01 4 
(0.03) 93,243 (0.03) 157,382 
Other course substitution 
CTE courses taken -0.02 3 0.02 3 
(0.05) 168,262 (0.07) 114,410 
Other courses taken -0.06 2 -0.02 3 
(0.12) 93,243 (0.12) 114,410 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard 
errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. 
Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth selection. N is the number 


of students. 
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Table 7. Correlation Between College Outcomes and Intervention Completion Status 


Take Pass credit- At least 60 
developmental bearing 15+ credits in credits in 3 
College type and subject course course in subject first term years 
2-year college, math 
No successful exit (bz) 0.083*** -0.038*** -0.023*** -0.021*** 
(0.01) (0.01) (0.01) (0.01) 
Successful exit (b,) -0.087*** 0.01 0.01 -0.016* 
(0.01) (0.01) (0.01) (0.01) 
p-value (bz = b,) 0.000 0.000 0.002 0.541 
Observations 11292 11292 11292 11292 
R-squared 0.054 0.088 0.038 0.066 
4-year college, math 
No successful exit (bz) 0.083*** -0.071*** -0.053*** -0.060*** 
(0.01) (0.01) (0.01) (0.01) 
Successful exit (b;) (0.02) -0.051*** 0.024* -0.030*** 
(0.01) (0.01) (0.01) (0.01) 
p-value (bz = b,) 0.000 0.098 0.000 0.019 
Observations 9296 9296 9296 9296 
R-squared 0.032 0.075 0.041 0.089 
2-year college, English 
No successful exit (bz) 0.041 *** -0.007 -0.017** -0.007 
(0.01) (0.01) (0.01) (0.01) 
Successful exit (b;) -0.082*** 0.02 0.01 (0.01) 
(0.01) (0.02) (0.01) (0.01) 
p-value (bz = b,) 0.000 0.072 0.051 0.856 
Observations 7694 7694 7694 7694 
R-squared 0.081 0.091 0.044 0.057 
4-year college, English 
No successful exit (bz) 0.038*** 0.030* -0.064*** -0.023 
(0.01) (0.02) (0.02) (0.02) 
Successful exit (b,) -0.054*** 0.01 0.02 (0.02) 
(0.01) (0.02) (0.02) (0.02) 
p-value (bz = b,) 0.000 0.314 0.000 0.941 
Observations 4789 4789 4789 4789 
R-squared 0.117 0.046 0.055 0.072 


Notes: ACT test year 2014 and 2015. Fixed effect for ACT test score. Additional controls for cubic function of test scores in 
other ACT subjects, race, gender, FRL eligibility, and whether the student was below cut points in any other subject. Only below 
cut score students included. Robust standard errors. Omitted group is students not in intervention. P-values indicate p-value of F- 
test of equality of the “No successful exit” and “Successful exit” coefficients. *, ** and *** denote statistical significance at the 
0.10, 0.05 and 0.01 levels, respectively. 
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Table 8. First-Stage Discontinuity in Complier Districts: Grade 8—9 Intervention 


Math English 


BW=2 BW=3 BW=4 BW=2 BW=3 BW=4 


Discontinuity in TI 
participation 


F-statistic 
N 


0.37%** 0.38*** —0.36*** ~—0.60*** = O.STHRE 0.554 
(0.06) (0.04) (0.04) (0.10) (0.07) (0.06) 


26.2 36.1 39.1 21.6 28.8 30.5 
1127 1623 1782 391 577 730 


Notes: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard 


errors in parentheses. Discontinuity in TI participation is estimated by regression TI participation on EXPLORE score, an 


indicator of falling below cutoffs, and the interaction of the two terms. Complier districts defined as districts with an estimated 
district-specific discontinuity of 0.20 with at least 50 students within bandwidth. Columns show bandwidths of 2, 3, and 4 points 
around the cutoff in each subject. The F-statistic tests the joint significance of the indicator variable of falling below the cutoff 
and its interaction with the EXPLORE score. The critical value is 11.59 based on Stock and Yogo (2005). 
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Table 9: Estimated Effect of Targeted Interventions (TI) in Grades 8 and 9 on Later Outcomes 
Take credit-bearing 


Take DE course college course in 
Above ACT Graduate high Attend in subject in first subject in first year 
Take ACT threshold ACT score school college year in college in college 

Math 

Discontinuity pre -0.01 -0.05 -0.22 -0.00 -0.02 -0.08*** 0.05 
(0.01) (0.03) (0.23) (0.01) (0.02) (0.03) (0.03) 

Discontinuity post -0.05* -0.06 -0.42 -0.06* -0.07 0.00 -0.04 
(0.03) (0.07) (0.47) (0.03) (0.04) (0.04) (0.06) 

Difference in RD -0.05 -0.01 -0.19 -0.05 -0.05 0.08* -0.09 
(0.03) (0.08) (0.51) (0.03) (0.05) (0.05) (0.07) 

Bandwidth 4 2 2 4 4 3 4 

N 8612 5213 5213 8612 8612 3875 4184 

English 

Discontinuity pre 0.03 -0.07 -0.33 -0.00 0.07* -0.04 0.04 
(0.03) (0.04) (0.30) (0.03) (0.04) (0.03) (0.05) 

Discontinuity post 0.02 -0.19** -0.90 -0.07 0.06 0.04 -0.20* 
(0.06) (0.08) (0.59) (0.06) (0.07) (0.04) (0.12) 

Difference in RD -0.02 -0.12 -0.57 -0.07 -0.01 0.08 -0.24 
(0.07) (0.09) (0.62) (0.06) (0.08) (0.06) (0.14) 

Bandwidth 4 

N 3415 2304 2927 3415 3415 1377 1377 


Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates 
assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth 
selection. N is the number of students. 


